System and method for identifying the existence and position of text in visual media content and for determining a subject&#39;s interactions with the text

ABSTRACT

A reading meter system and method is provided for identifying the existence and position of text in visual media content (e.g., a document to be displayed (or being displayed) on a computer monitor or other display device) and determining if a subject has interacted with the text and/or the level of the subject&#39;s interaction with the text (e.g., whether the subject looked at the text, whether the subject read the text, whether the subject comprehended the text, whether the subject perceived and made sense of the text, and/or other levels of the subject&#39;s interaction with the text). The determination may, for example, be based on data generated from an eye tracking device. The reading meter system may be used alone and/or in connection with an emotional response tool (e.g., a software-based tool for determining the subject&#39;s emotional response to the text and/or other elements of the visual media content on which the text appears). If used together, the reading meter system and emotional response tool advantageously may both receive, and perform processing on, eye date generated from a common eye tracking device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/089,200, filed Aug. 15, 2008, which is hereby incorporatedby reference herein in its entirety.

FIELD OF THE INVENTION

The invention relates to a system and method for identifying theexistence and position of text in visual media content, and fordetermining whether a subject has interacted with the text and/or thelevel of the subject's interaction with the text based on, for example,data generated from an eye tracking device.

BACKGROUND OF THE INVENTION

Systems for determining whether a user has looked at text on a displayare generally known. These systems, however, have a number oflimitations and drawbacks. For example, there is a difference betweensimply determining whether a user's eyes have passed over text, anddetermining whether the user has interacted with the text and, if so,the level of the subject's interaction with the text (e.g., whether theuser actually read and/or comprehended the text).

Additionally, while some text identification tools exist, these toolstoo suffer from various limitations and drawbacks.

SUMMARY OF THE INVENTION

The invention addressing these and other drawbacks in the art relates toa reading meter system and method for identifying the existence andposition of text in visual media content (e.g., a document or othervisual media content to be displayed (or being displayed) on a computermonitor or other display device) and determining whether a subject hasinteracted with the text and/or the level of the subject's interactionwith the text (e.g., whether the subject looked at the text, whether thesubject read the text, whether the subject comprehended the text,whether the subject perceived and made sense of the identified text,and/or other levels of the subject's interaction with the text). Thedetermination may, for example, be based on eye data generated from aneye tracking device. Eye data may include, but not be limited to, pupildata, blink data, gaze data, eye position/movement, pupil dilation,and/or other eye data.

The reading meter system may be used alone and/or in connection with anemotional response tool (e.g., a software-based tool for determining thesubject's emotional response to the text and/or other elements of thevisual media content on which the text appears). If used together, thereading meter system and emotional response tool advantageously may bothreceive, and perform processing on, eye data generated from a common eyetracking device.

According to one implementation of the invention, the reading metersystem may comprise a general purpose computer programmed with a readingmeter software application and/or a software-based emotional responsetest, an eye tracking device, a computer monitor or other display device(or virtual display), one or more input devices, one or more outputdevices, and/or other system components.

The reading meter software application may comprise one or more of atext identification and position determination module, an eye gazepattern determination module, a text interaction determination module, acognitive workload determination module, a memory impact determinationmodule, and/or other modules as described herein.

The text identification and position determination module may compriseone or more sub-modules for identifying the existence of text in visualmedia content, identifying the position of the identified text, andidentifying geometry (or geometrical) characteristics of the identifiedtext, and/or for performing other functions.

The text identification module may identify some or all of the text invisual media content. The text may be identified based on blocks of textand/or portions thereof such as, for example, paragraphs, sentences,phrases, words, and/or other portions.

The text position identification module may identify the position (e.g.,x-y or other coordinates) of an identified portion of the text inrelation to the visual media content in which it appears (e.g., adocument that includes both text and other display elements).

The text geometry characteristics module may identify variouscharacteristics (or attributes) of identified text including, but notlimited to, text character height, text character width, text characterfont, number of letters, number of words, length of words, length oflines of text, number of lines of text, etc. Various Optical CharacterRecognition (OCR) (or other) techniques may be implemented by the textgeometry characteristics module to identify one or more of theaforementioned characteristics.

The identification of a text portion and its position may be performedmanually, automatically, and/or semi-automatically. The textidentification and position determination module may direct the storageof information regarding the identified portions of text, theirrespective positions in a given piece of visual media content, and/orthe characteristics of the identified portions of text. Such informationmay be stored in a text identification and position database, or otherstorage mechanism.

According to an aspect of the invention, the eye gaze patterndetermination module may be operable to determine a subject's eye gazepattern based on collected eye gaze data. For example, the eye gazepattern determination module may receive eye gaze data (for one or botheyes) from an eye tracking device, wherein such data indicates theposition (e.g., x-y or other coordinates) on a computer (or other)display at which a subject's eye(s) looked at a particular sample time.Based on a time series of such eye position data (e.g., at apredetermined sampling rate), the eye gaze pattern determination modulecan determine the subject's eye gaze pattern (e.g., the positions of thevisual media content at which the user looked over some time) inrelation to the coordinates of the visual media content being displayed.The eye gaze pattern determination module may direct the storage of eyegaze pattern information for a subject's interaction with a given pieceof visual media content. Such information may be stored in an eye gazepattern database, or other storage mechanism.

The eye gaze pattern determination module may account for the distancebetween the subject's eye(s) and the displayed visual media content, asthis distance affects a subject's gaze pattern while reading. Thisdistance may be measured automatically, or manually and input by thesubject or a test administrator. The focal vision is the center ofvision and covers a circle of ca. two degrees around the gaze point.Focal vision allows the subject to see clearly and to read. When anobject is moved away from the eye, for example, the focal vision coversa larger area of the object on the cost of resolution, that is, theretinal image becomes smaller. This influences the saccade length andthe ability and ease of reading a text. Reading saccades span, onaverage, about two degrees of visual angle, although this can be betterexpressed in terms of a span of 7 to 9 letter spaces, since the numberof letters covered remains largely invariant despite differences in textsize or distance.

The eye gaze pattern determination module may receive data from the textidentification and position determination module to account for changesin, for example, text block sizes, font sizes, word length, and wordcomplexity, among other things, as these may affect a subject's gazepattern while reading.

The text interaction determination module may process information (e.g.,the text identification/position information, the eye gaze patterninformation, and/or other information) to determine, for example,whether the subject has interacted with the text in the visual mediacontent, and/or the level of the subject's interaction with the text.For example, upon processing some or all of the foregoing information,if the text interaction determination module determines that at leastsome of the coordinate positions of an identified text block in thevisual media content coincide with at least some of the coordinatepositions of a subject's gaze (e.g., based on eye gaze patterninformation), the text interaction determination module may determinethat the subject has interacted with the identified text. Based on this,and/or other processing, additional information can be determined. Forexample, the eye gaze pattern of the subject may be effectivelysuperimposed on to the visual media content to determine informationregarding the subject's level of interaction with the identified text.Information regarding the subject's level of interaction with theidentified text may include, for example, whether the subject looked atthe identified text, whether the subject read the identified text,whether the subject comprehended the identified text, whether thesubject perceived and made sense of the identified text, and/or otherlevels of interaction. The text interaction determination module maydirect the storage of information regarding the subject's interactionswith the identified text in a given piece of visual media content. Suchinformation may be stored in a text interaction database, or otherstorage mechanism.

The text interaction determination module may comprise one or moresub-modules for reading determination, comprehension determination,and/or other functions. The reading determination module may determineif the subject read the identified text. The comprehension determinationmodule may determine if the subject comprehended the identified text.

According to one aspect of the invention, the cognitive workloaddetermination module may monitor the environment, stimulus (e.g.,document, visual media content, and/or other stimulus), and/or subjectto determine workload on the subject continuously to provide informationon when the subject has the spare capacity to receive and comprehend thetext during text interaction. The workload determination may be based onpupilometrics and/or gaze patterns.

According to one aspect of the invention, the memory impactdetermination module may receive data from the cognitive workloaddetermination module and the eye gaze pattern determination module toidentify current words of interest which attract special attention(e.g., direction of sight, longer duration of fixations, or returning toparticular parts of the text). Since the reading process is mainlycognitive, there is a direct link between gaze patterns while readingand the processing in working memory.

In one implementation of the invention, the emotional response tool mayinclude a software application running on the same (or another computer)as the reading meter software application. The emotional response toolsoftware application may include an emotional response determinationmodule, a visual attention determination module, and/or other modules.

The emotional response determination module may determine informationregarding the subject's emotional response to the visual media content,or portions of the visual media content (e.g., one or more portions ofthe identified text, images and/or other portions of the visual mediacontent). The emotional response may include, for example, the direction(valence) and magnitude (arousal) of any emotional response.

The visual attention determination module may determine visual attentioninformation for the subject. Visual attention information may include,for example, information regarding points or areas of the visual mediacontent on which the subject focused for at least a minimum amount oftime, information regarding points or areas of the visual media contenton which the subject re-focused on or returned to (e.g., return points),and/or other information. The visual attention information may indicatethe points or areas of the visual media content that drew and/or heldthe subject's attention. The emotional response determination module andvisual attention determination module may direct the storage ofinformation regarding the subject's emotional response and visualattention. Such information may be stored in an emotional response andvisual attention database, or other storage mechanism.

According to one implementation of the invention, in operation, visualmedia content may be displayed to a subject (or user) via a computermonitor or other display device. The visual media content may comprise astimulus or other piece(s) of content, in any format (e.g., on a slide),which may include various display elements, at least a portion of whichmay include text.

Assuming the visual media content includes a portion of text, the textidentification and position determination module may identify theexistence and position of text (absolute or relative to other displayelements) in the visual media content at any time (e.g., when presented,or before or after presentation). Various eye property data from asubject positioned before the display may be acquired (e.g., by an eyetracking device associated with the monitor). The eye data may becollected at a predetermined sampling rate. The eye data may include eyeposition data, eye blink data, pupil dilation data and/or other eyedata. Other physiological data, and/or other data may be collected(e.g., by a data collection module). Data concerning one or morephysiological attributes of the subject may be collected from one ormore emotion detection sensors, and/or one or more environmentalparameters (e.g., light intensity, noise, temperature, and/or otherparameters) may also be measured and collected.

An exemplary (and non-limiting) description of the set-up andcalibration of an eye tracking device and other sensors (along with datacollection and analysis) is described in detail in U.S. PatentApplication Publication No. 2007/0066916 A1, published Mar. 22, 2007,which is hereby incorporated herein by reference in its entirety.

The eye data collected (and/or other collected information), may beused, for example, by the eye gaze pattern determination module todetermine the eye gaze pattern(s) of the subject. The textidentification and position data, the eye gaze pattern data, and/orother collected data may be provided as input to the text interactiondetermination module. The text interaction module may determine whetherthe subject has interacted with the identified text and/or the level ofthe subject's interactions with the text (e.g., whether the subjectlooked at the identified text, whether the subject read the identifiedtext, whether the subject comprehended the identified text, whether thesubject perceived and made sense of the identified text, and/or otherlevels of interaction), and/or other portions of the display. Ifdesired, visual attention and/or emotional response may also bedetermined. The results of the analysis may be stored and/or output in avariety of formats.

Various other objects, features, and advantages of the invention will beapparent through the detailed description of the preferred embodimentsand the drawings attached hereto. It is also to be understood that boththe foregoing general description and the following detailed descriptionare exemplary and not restrictive of the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary illustration of a reading meter system, accordingto an aspect of the invention.

FIG. 2 is a schematic block diagram illustrating exemplary (andnon-limiting) features and functionality of a text identification andposition determination module and an eye gaze pattern determinationmodule, as well as the communication there-between.

FIG. 3 is an exemplary illustration of an eye gaze pattern for anexample wherein a user reads text (reading example), and an eye gazepattern for an example wherein a user skims text (skimming example).

FIG. 4 is an exemplary illustration of timeline when processing visualinformation, according to an aspect of the invention.

FIG. 5 is an exemplary illustration of a flowchart of processingoperations, according to an aspect of the invention.

FIGS. 6A to 6D depict exemplary output charts/plots that may bepresented, according to an aspect of the invention.

DETAILED DESCRIPTION OF THE INVENTION

According to one implementation of the invention, as shown for examplein FIG. 1, a reading meter system 100 may comprise a general purposecomputer 110 programmed with a reading meter software application 120,an eye tracking device 160, a computer monitor or other display device(or virtual display) 162, one or more input devices 170, one or moreoutput devices 180, one or more and/or other components. Computer 110may comprise a processor (not shown), circuitry and/or other hardwareoperable to execute computer-readable instructions. According to anaspect of the invention, computer 110 may include one or morecomputer-readable storage media configured to store one or more softwaremodules, wherein the software modules include computer-readableinstructions that when executed by the processor cause the processor toperform the functions described herein.

The reading meter software application 120 may comprise one or moresoftware modules that enable various features and functions of theinvention. Non-limiting examples of the software modules may include oneor more of a text identification and position determination module 122,an eye gaze pattern determination module 124, a text interactiondetermination module 126, a cognitive workload determination module 144,a memory impact determination module 148, and/or other modules.

The text identification and position determination module 122 maycomprise one or more sub-modules for text identification 122 a, textgeometry characteristics 122 b, text position identification 122 c,and/or for performing other functions. The text identification module122 a may identify some or all of the text in visual media content. Thetext may be identified based on blocks of text and or portions thereofsuch as, for example, paragraphs, sentences, phrases, words, and orother portions.

Various known text identification techniques may be used by textidentification module 122 a to identify text in visual media content.For example, the text identification module 122 a may use the imageprocessing techniques discussed in article titled “Text Detection andCharacter Recognition using Fuzzy Image Processing”, by Alata et al.,which is hereby incorporated by reference herein in its entirety, todetect and recognize text in a document image.

Text position identification module 122 c may identify the position(e.g., x-y or other coordinates) of an identified portion of the text inrelation to the visual media content in which it appears (e.g., adocument that includes both text and other display elements).

The text geometry characteristics module 122 b may identify variouscharacteristics (or attributes) of identified text including, but notlimited to, text character height, text character width, text characterfont, number of letters, number of words, length of words, length oflines of text, number of lines of text, etc. As previously noted,various Optical Character Recognition (OCR) (or other) techniques may beimplemented by text geometry characteristics module 122 b to identifyone or more of the aforementioned characteristics.

The identification of a text portion and its position may be performedmanually, automatically, and/or semi-automatically. The textidentification and position determination module 122 may direct thestorage of information regarding the identified portions of text, theirrespective positions in a given document, and/or the characteristics ofthe identified portions of text. Such information may be stored in atext identification and position database 192, or other storagemechanism.

The eye gaze pattern determination module 124 may be operable todetermine a subject's eye gaze pattern based on collected eye gaze data(e.g., generated by eye tracking device 160). For example, the gazepattern determination module may receive eye gaze data (for one or botheyes) from eye tracking device 160, wherein such data indicates theposition (e.g., x-y or other coordinates) on a computer (or other)display at which a subject's eye(s) looked at a particular sample time.Based on a time series of such eye position data (e.g., at apredetermined sampling rate), the eye gaze pattern determination module124 can determine the subject's eye gaze pattern (e.g. the positions ofthe document at which the user looked over some time) in relation to thecoordinates of the document being displayed. The eye gaze patterndetermination module 124 may direct the storage of eye gaze patterninformation for a subject's interaction with a given piece of visualmedia content. Such information may be stored in an eye gaze patterndatabase 193, or other storage mechanism.

The eye gaze pattern determination module may account for the distancebetween the subject's eye(s) and the displayed visual media content, asthis distance affects a subject's gaze pattern while reading. Thisdistance may be measured automatically, or manually and input by thesubject or a test administrator. The focal vision is the center ofvision and covers a circle of ca. two degrees around the gaze point.Focal vision allows the subject to see clearly and to read. When anobject is moved away from the eye, for example, the focal vision coversa larger area of the object on the cost of resolution, that is, theretinal image becomes smaller. This influences the saccade length andthe ability and ease of reading a text. Reading saccades span, onaverage, about two degrees of visual angle, although this can be betterexpressed in terms of a span of 7 to 9 letter spaces, since the numberof letters covered remains largely invariant despite differences in textsize or distance.

The eye gaze pattern determination module 124 may receive data from thetext identification and position determination module 122 to account forchanges in, for example, text block sizes, font sizes, word length, andword complexity, among other things, as these may affect a subject'sgaze pattern while reading. FIG. 2 is a schematic block diagramillustrating exemplary (and non-limiting) features and functionality oftext identification and position determination module 122 and eye gazepattern determination module 124, as well as the communicationthere-between.

The eye gaze pattern determination module 124 may receive as input oneor more properties/characteristics of the identified text from textidentification and position determination module 122 to optimize thegaze analysis. The one or more properties may include, but not belimited to, text location, text size, character size, text box size,length of words, text complexity, number of lines of text, verticalinterval between lines, color of text, contrast, font type, orientation(x, y, z, t in dynamic content), lix number (readability index),language (using for example input from a dictionary), start/end ofsentences, and/or other text properties. The input may be utilized todistinguish, for example, reading from skimming as skimming could beinterpreted as reading longer words or words with greater horizontaldistance. The performance of the system may be improved by adjustingweights/thresholds of the pattern analysis to the given text properties.

According to an aspect of the invention, text identification andposition determination module 122 may provide properties associated withthe visual media content as input to eye gaze pattern determinationmodule 124. Based on the input, it may be possible to distinguishbetween different text types, distinguish non-text portions from textportions, and/or perform other determinations, thereby improving gazeanalysis and preventing non-text portions to be classified as read. Forexample, in some cases gaze patterns associated with non-text portionsof visual media content may be similar to reading patterns which wouldgive a false positive. In these cases, providing properties associatedwith visual media content as input to the eye gaze pattern determinationmodule 124 may optimize gaze analysis. In order to support the textidentification and position determination module 122 in textrecognition, a preliminary output from eye gaze pattern determinationmodule 124 may be used to indicate where there is a good chance offinding text because gaze patterns look like reading. According to anaspect of the invention, the preliminary output from eye gaze patterndetermination module 124 may be used to indicate whether the subjectgazed at an area of the visual media content. Text identification andposition determination module 122 may perform text recognition in areasindicated by eye gaze pattern determination module 124.

The text interaction determination module 126 may process the textidentification/position information and the eye gaze pattern informationto determine, for example, whether the subject has interacted with thetext in the visual media content, and/or the level of the subject'sinteraction with the text. For example, by processing the foregoingdata, if the text interaction determination module 126 determines thatthe coordinate positions of identified text in the document coincidewith at least some of the coordinate positions of a subject's gaze(e.g., based on eye gaze pattern information), the text interactiondetermination module 126 may determine that the subject has interactedwith the identified text. Based on this, and/or other processing,additional information can be determined. For example, the eye gazepattern of the subject may be effectively superimposed on to the visualmedia content to determine information regarding the subject's level ofinteraction with the identified text. Information regarding thesubject's level of interaction with the identified text may include,whether the subject looked at the identified text, whether the subjectread the identified text, whether the subject comprehended theidentified text, whether the subject perceived and made sense of theidentified text, and/or other levels of interaction. FIG. 3 is anexemplary illustration of an eye gaze pattern for an example wherein auser reads text (reading example), and an eye gaze pattern for anexample wherein a user skims text (skimming example).

The text interaction determination module 126 may direct the storage ofinformation regarding the subject's interactions with the identifiedtext in a given document. Such information may be stored in a textinteraction database 194, or other storage mechanism.

If the text interaction determination module 126 determines that, forexample, at least a portion of the subject's eye gaze pattern overlapswith the coordinate positions of identified text in the document, thetext interaction determination module 126 may determine that the subjecthas interacted with the identified text, at least to some extent. Thismay mean that, at a minimum, the subject looked at the identified text.

The text interaction determination module 126 may comprise one or moresub-modules for reading determination 126 a, comprehension determination126 b, and/or for performing other functions. The reading determinationmodule 126 a may determine whether the subject read the identified text.Various techniques (e.g., reading pattern templates) may be used byreading determination module 126 a to determine whether the subject readthe identified text.

According to one aspect of the invention, a subject's eye gaze patternmay be analyzed to determine if the subject's eye movements correlatewith a reading pattern. As one example, in many languages a readingpattern may include eye movement from left to right, then back to theleft and down (e.g., at the end of a line). If there is a correlationbetween a stored reading pattern and the subject's eye movements, it maybe determined that the subject has read the identified text. Becausepeople read at different rates, subject profile information may be usedto calibrate and/or normalize data.

According to an aspect of the invention, attributes relating to asubject's interaction with the text (e.g., speed, acceleration, anddirection of eye movements) may, in part, be used to determine if thesubject read the identified text. A subject's eye gaze patterninformation may be analyzed to determine the speed at which the subjectinteracted with the text. For example, quick eye movements (microsaccades) may indicate that the subject only skimmed through the textand did not read it.

The comprehension determination module 126 b may determine whether thesubject comprehended the identified text. The speed at which the subjectinteracted with the text, and/or the uniformity of the speed at whichthe subject interacted with the text may, in part, also determine if thesubject comprehended the text or had difficulty comprehending the text.For example, while reading the text at a certain speed, if the subjecthad to slow down or revisit one or more portions of text (i.e. the speedat which the subject is interacting with text decreases), it may bedetermined that the subject had difficulty comprehending the text orportions thereof.

According to one aspect of the invention, the subject's emotionalresponse information may also be used, either alone or in combinationwith the speed information, to determine if the subject comprehended thetext. For example, the emotional response of the subject may indicatethat the subject is confused or irritated, which in turn may indicatethat the subject had difficulty comprehending the text.

The eye tracking device 160 may include a camera or other knowneye-tracking device that records and tracks data relating to various eyeproperties of the subject. Examples of eye property data that may becollected may include eye position data, eye blink data, pupil dilationdata, and/or other eye data.

Display 162 may comprise a physical display (e.g., a computer monitorcomprising one or more Cathode Ray Tube (CRT) displays, digital flatpanel displays or other display devices) or a virtual display (e.g., amulti-screen chamber like the CAVE sold by Fakespace Systems Inc.) forpresenting visual instructions and messages, documents (e.g., slides orstimuli which may include various display elements, at least a portionof which may include text), and/or other information to subjects.

One or more input devices 170 may comprise one or more of manual inputdevice(s) 172, sensors(s) 174, and/or other input devices 176 to receiveinput (e.g., from subjects). Manual input device(s) 172 may include oneor more of a keyboard, mouse, and/or other input device that may enablesubjects to manually input data. Sensor(s) 174 may include one or moreemotional detection sensors, environmental sensors, and/or othersensors.

Emotion detection sensors may comprise, for example, one or morephysiological sensors such as galvanic skin response sensors, facialrecognition sensors, and/or other emotion detection sensors that maydetect various physiological responses from subjects.

Environmental sensors may comprise, for example, one or more lightintensity sensors, background noise sensors, temperature sensors, and/orother sensors that may measure various environmental parameters.

One or more output devices 180 may include one or more of speaker 182,and/or other output devices 184 (e.g., a printer). Speaker 182 maycomprise one or more speakers for audible reproduction of, for example,audio instructions or messages, and/or other information to subjects.

According to one aspect of the invention, the reading meter softwareapplication 120 may further comprise one or more of an initial setupmodule 128, a content presentation module 130, a data collection module132, an output module 134, an interface controller module 136, and/orother modules 140.

The initial setup module 128 may perform various setup/calibrationfunctions. Examples of these functions may include, among other things,test/visual media content setup, subject setup, various calibrationfunctions, or other functions. A test including one or more pieces ofvisual media content to be presented to the subject may be selected viathe initial setup module 128. The one or more pieces of visual mediacontent may include various types of elements, including text.

Visual media content presentation properties may be selected by initialsetup module 128. For example, for a given test, one or more of theduration of presentation for various pieces of content may be selectedalong with the order of presentation of content, whether any contentshould be simultaneously presented, and/or other content presentationproperties. Output presentation properties, for example, outputpresentation format, amount of information to be presented, and/or otheroutput presentation properties, may be specified by initial setup module128. Who should receive the output, how should the output be received,etc. may also be specified.

Various profile information regarding a subject may be collected by theinitial setup module 128 including, but not limited to, name, age,gender, and/or other profile information. Various calibration protocolsmay be implemented by the initial setup module 128 including, forexample, one or more calibration protocols for adjusting various sensorsto an environment, adjusting various sensors to a subject within theenvironment, and determining a baseline emotional level for the subjectwithin the environment.

Additional details on these and other calibration and initial setupfunctions that may be performed by initial setup module 128 arediscussed in U.S. application Ser. No. 11/522,476, entitled “System andMethod for Determining Human Emotion by Analyzing Eye Properties,” filedSep. 18, 2006, and in U.S. application Ser. No. 12/170,059, entitled“System and Method for Calibrating and Normalizing Eye Date in EmotionalTesting,” filed on Jul. 9, 2008, the disclosures of which are herebyincorporated by reference in their entireties.

The initial setup module 128 may direct storage of the setup informationin setup database 191, or other storage mechanism.

The content presentation module 130 may facilitate the presentation ofthe visual media content.

The data collection module 132 may govern the collection of various eyeproperty data from eye tracking device 160. The eye property data mayinclude eye position data, eye blink data, pupil dilation data, and/orother eye data.

Data collection module 132 may further govern the collection of variousphysiological data, and/or other data. Data concerning one or morephysiological attributes of the subject may be collected from one ormore emotion detection sensors, and/or other sensors.

Data collection module 132 may collect various environmental parameters(e.g., light intensity, noise, temperature, and/or other environmentalparameters) that may be measured by one or more environment sensors. Thedata collection module 132 may direct storage of the collected data in acollected data database 195.

In one implementation, the output module 134 may selectively enablevarious types of output to the one or more output devices 180. Forexample, output module 134 may be used to produce reports based on theresults of various analyses of data described herein. Various electronicand/or printed output types may be used to present the results. Theseoutput types may include representation in the form of graphs, text,illustrations, gaze plots, audio, and/or video playback, or other outputtypes.

FIGS. 6A to 6D depict exemplary outputs that may be presented by outputmodule 134. The number of subjects who have read the identified text outof a total number of subjects may be indicated by “Readers”. The averageamount of identified text read by the subjects may be indicated by Read(%). For example, the number of subjects who read the identified text ofFIG. 6A is 10 out of a total number of 13 subjects. Also, the averageamount of text read by the subjects is 53%. FIGS. 6A to 6D depict aReading Intensity Map which may illustrate both what has been read (byone or more subjects) and the reading intensity. Reading intensity is ameasure of how much time subjects spent reading a particular part of thetext. An area which attracted more than 60% of the reading time may bemarked as high reading intensity area (which is depicted as the darkestshade of gray in the figures). Most subjects have read the text in thisarea, and it may be an indicator that either the text is very attentiongrabbing, or unusual or difficult to read which is why subjects havespent much time there. Medium reading intensity depicted as a lightershade of gray in the figures (lighter than the high reading intensityshade but darker than a low reading intensity shade) marks an area where30-60% of the time was spent. These areas may have been read by aroundhalf the subjects. Low reading intensity areas have attracted less than30% of the reading time, and are depicted by the lightest shade of grayin the figures. These areas may have only been read by a couple ofsubjects.

It may be seen in FIG. 6A, for example, that the words “SloopyBucks”,“UWeekly” and “UW” have achieved a high reading intensity. These are alluncommon words, and thus subjects have dwelled longer on these words. InFIG. 6B, the portion “10 days of text ads per month” is bold text, whichgrabbed the subjects' attention, and raised the reading intensity.Reading intensity is normalized in each figure by the ratio of readersto the total number of subjects. This means that an area which has veryfew readers may have no Intensity map, as depicted in FIG. 6D forexample. This may be helpful to easily identify the most and least readparts of visual media content. The numbers (e.g., 1, 2, 3, and 4)assigned to the text in FIGS. 6A to 6D may depend on the number ofreaders and/or how much they read.

The interface controller module 136 may be associated with and/or incommunication with the one or more input devices, the one or more outputdevices, and/or other devices.

According to one aspect of the invention, reading meter system 100 maybe used alone and/or in connection with emotional response tool 150. Ifused together, reading meter system 100 and emotional response tool 150advantageously may both receive, and perform processing on, eye datagenerated by a common eye tracking device (e.g., eye tracking device160). The emotional response tool 150 may include an emotional responsetool software application 152 running on the same (or another) computer110. The emotional response tool software application 152 may includeone or more software modules that enable various features and functionsof the invention. Non-limiting examples of the software modules mayinclude one or more of an emotional response determination module 152 a,a visual attention determination module 152 b, and/or other modules.

In various implementations, information determined from one or moremodules of emotional response tool software application 152 may be usedas input for one or more processing operations performed by one or moremodules of reading meter application 120, and vice versa.

In one implementation, the emotional response determination module 152 amay determine information regarding the subject's emotional responses tovisual media content, one or more portions of identified text, and/orother portions of a given piece of visual media content. Emotionalresponse determination module 152 a may analyze and process the eye datafrom eye tracking device 160 to determine the emotional response. Theemotional response may include, for example, the direction (valence) andmagnitude (arousal) of any emotional response.

The visual attention determination module 152 b may determine visualattention information for the subject. Visual attention information mayinclude, for example, information regarding points or areas of thedocument on which the subject focused for at least a minimum amount oftime, information regarding points or areas of the documents on whichthe subject re-focused on or returned to (e.g., return points), and/orother information. The visual attention information may indicate pointsor areas of the visual media content that drew and/or held a subject'sattention. Visual attention determination module 152 b and/or textinteraction determination module 126 may measure an amount of time oneor more subjects spent reading a particular part of the text which maybe referred to as reading intensity. The emotional responsedetermination module 152 a and visual attention determination module 152b may direct the storage of information regarding the subject'semotional response and visual attention. Such information may be storedin an emotional response and visual attention database 196, or otherstorage mechanism.

Additional details regarding the emotional response determination andvisual attention determination are disclosed in U.S. application Ser.No. 11/522,476, filed Sep. 18, 2006; U.S. application Ser. No.11/685,552, filed Mar. 13, 2007; U.S. application Ser. No. 12/170,059,filed Jul. 9, 2008; and U.S. application Ser. No. 12/170,041, filed Jul.9, 2008, which are incorporated herein by reference in their entirety.

According to an aspect of the invention, cognitive workloaddetermination module 144 may monitor the environment, stimulus, and/orsubject to determine workload on the subject continuously to provideinformation on when the subject has the spare capacity to receive andcomprehend the text during text interaction. The workload determinationmay be based on pupilometrics and/or gaze patterns.

Memory impact determination module 148 may receive data from thecognitive workload determination module and the eye gaze patterndetermination module to identify current words of interest which attractspecial attention (e.g., direction of sight, longer duration offixations, or returning to particular parts of the text). Since thereading process is mainly cognitive, there is a direct link between gazepatterns while reading and the processing in working memory.

According to an aspect of the invention, cognitive workloaddetermination module 144 and/or memory impact determination module 148may determine whether a subject perceived or made sense of theidentified text.

FIG. 4 is an exemplary illustration of timeline when processing visualinformation and how it influences behavior, according to an aspect ofthe invention. The timeline includes one or more steps, for example,pre-attention, pre-cognition, cognitive effect, behavioral effect,and/or other steps. These steps may be influenced and guided byemotions. In the pre-attention step, for example, a stimulus (forexample, visual media content) may initiate certain physiological andchemical changes in the body of a subject, These changes may influencelow-level attention, and initiate emotions that may be triggered in thebody together with the particular physiological changes that belong tothose emotions, The emotions may guide selective attention. Low-levelattention is automated and may subconsciously scan the whole visualfield and spot the eye-catchers-the visual elements that catchattention. This is the part of attention that is automatic, involuntaryand subconscious driven by emotions. Selective attention moves like aspotlight from one area of the visual field to another, analyzing theelements in more detail. Selective attention moves according to someattention values being calculated from partly the stimulus itself andpartly by what is in the subject's mind, including emotions. Emotionsparticipate in guiding and qualifying visual attention.

Together with the result of the selective attention, the emotions andphysiological processes initiate pre-cognition. Pre-cognition mayinclude non-verbal information, the potential for thoughts and images,and/or other information. Cognition, emotions and their respectivephysiological changes manifest into cognition and feelings. Feelings arepart of emotions and physiological processes that become conscious.Cognition (thinking) and feelings are conscious phenomena that mayinfluence behavior (for example, buying behavior, and/or otherbehavior). Behavior is, in other words, influenced by both subconsciousemotions, cognition and feelings.

FIG. 5 illustrates an exemplary flowchart 200 of processing operations,according to an aspect of the invention. The described operations may beaccomplished using some or all of the system components described indetail above and, in some implementations, various operations may beperformed in different sequences. In other implementations, additionaloperations may be performed along with some or all of the operationsshown in FIG. 5. In yet other implementations, one or more operationsmay be performed simultaneously. Accordingly, the operations describedare exemplary in nature and, as such, should not be viewed as limiting.

In an operation 202, various preliminary (or setup) operations may beperformed (e.g., performed by initial setup module 128).

In an operation 204, visual media content may be displayed to a subject(or user) (e.g., via display 162). The visual media content may includevarious display elements, at least a portion of which may include text.

Assuming that the visual media content includes at least a portion oftext, the existence of text may be identified in an operation 206, andthe position of the text (e.g., absolute or relative to other displayelements) in the visual media content may be determined in an operation208. Characteristics of the text (e.g., text character height, textcharacter width, text character font, number of letters, number ofwords, length of words, length of lines of text, number of lines oftext, etc.) may also be determined in either or both of operation 206and 208. In one implementation, operations 206 and 208 may be performed,for example, by the text identification and position determinationmodule 122.

The text identification and position determination module 122 mayperform operations 206 and/or 208 when the visual media content ispresented, or before or after presentation. As detailed below, in oneimplementation, data generated regarding a subject's interactions withthe identified text (e.g., by text interaction determination module 126)and/or subject's emotional response to the identified text (e.g., byemotional response determination module 152 a) may be provided back tothe text identification and position determination module 122 as inputto refine the text identification and position determination.

In an operation 210, various eye property data may be acquired from asubject positioned before display 162 (e.g., by eye tracking device160). Collection of the eye property data may be governed by datacollection module 132. The eye data, collected at a predeterminedsampling rate, may include eye position data, eye blink data, pupildilation data, and/or other eye data. Other physiological data, and/orother data from the subject may also be acquired in operation 210 (andgoverned by data collection module 132). Data concerning one or morephysiological attributes of the subject may be collected from one ormore emotion detection sensors. Data indicative of one or moreenvironmental parameters (e.g., light intensity, noise, temperature, orother parameters) may also be acquired.

In an operation 212, the eye data collected (and/or other collectedinformation) may be utilized, for example, by the eye gaze patterndetermination module 124 to determine the eye gaze pattern of thesubject.

The text identification and position data, the eye gaze pattern data,and/or other collected data, may be provided as input to the textinteraction determination module 126. The text interaction determinationmodule 126 may, in an operation 214, make a determination as to whetherthe subject has interacted with the identified text. If a determinationis made that the subject interacted with the identified text at somelevel, processing may continue at operation 216. If a determination ismade that the subject did not interact with the text, some remedialaction or other action may be taken.

In an operation 216, the text interaction determination module 126 maydetermine whether the subject looked at the identified text. If the textinteraction determination module 126 determines that, for example, atleast a portion of the subject's eye gaze pattern overlaps with thecoordinate positions of identified text in the document, the textinteraction determination module 126 may determine that the subject hasinteracted with the identified text, at least to some extent. This maymean that, at a minimum, the subject looked at the identified text, andprocessing may continue with operation 218. If a determination is madethat the subject did not look at the text, some remedial action or otheraction may be taken.

In operation 218, a determination may be made as to whether the subjectread the identified text (e.g., by reading determination module 126 a asdescribed above). If a determination is made that the subject read thetext, processing may continue at operation 220. If a determination ismade that the subject did not read the text, some remedial action orother action may be taken.

In an operation 220, a determination may be made as to whether thesubject comprehended the identified text (e.g., by comprehensiondetermination module 126 b). If a determination is made that the subjectcomprehended the text, processing may continue at operation 222. If adetermination is made that the subject did not comprehend the text, someremedial action or other action may be taken.

According to an aspect of the invention, if desired, visual attentionand/or emotional response information may also be determined for asubject that has viewed (or is currently viewing) visual media content.

In an operation 222, visual attention information for the subject may bedetermined (e.g., by visual attention determination module 152 b).

Information regarding the subject's emotional responses to the visualmedia content, one or more portions of the identified text, and/or otherportions of the visual media content may be determined in an operation224 (e.g., by emotional response determination module 152 a).

According to an aspect of the invention, in an operation 226, feedbackregarding the subject's interactions with the identified text (e.g., bytext interaction determination module 126) and/or the subject'semotional response to the identified text (e.g., emotional responsedetermination module 152 a) may be provided to the text identificationand position determination module 122 to refine the text identificationand position determination.

For example, if it is determined in operation 218 that the subject hasread various text portions in the document, this information mayindicate the positions of the text versus non-text (e.g. images, and/orother non-text portions) portions in the document. Such information maybe provided as feedback to the text identification and positiondetermination module 122 to further refine the text identification andposition determination.

Although FIG. 5 depicts feedback being provided after processing ofoperation 226, it will be readily appreciated that the feedback may beprovided after performing one or more of operations 212, 214, 216, 218,220, 222, and/or 224.

In an operation 228, it may be determined if additional visual mediacontent is to be presented to the subject. If it is determined thatadditional visual media content is to be presented to the subject,processing may resume at operation 204. If it is determined that noadditional visual media content is to be presented to the subject,processing may end at operation 230.

Implementations of the invention may be made in hardware, firmware,software, or various combinations thereof. The invention may also beimplemented as computer-readable instructions stored on acomputer-readable storage medium which may be read and executed by oneor more processors. A computer-readable storage medium may includevarious mechanisms for storing information in a form readable by acomputing device. For example, a computer-readable storage medium mayinclude read only memory, random access memory, magnetic disk storagemedia, optical storage media, flash memory devices, and/or other storagemediums. Further, firmware, software, routines, or instructions may bedescribed in the above disclosure in terms of specific exemplary aspectsand implementations of the invention, and performing certain actions.However, it will be apparent that such descriptions are merely forconvenience, and that such actions may in fact result from computingdevices, processors, controllers, or other devices executing firmware,software, routines or instructions.

Other implementations, uses and advantages of the invention will beapparent to those skilled in the art from consideration of thespecification and practice of the invention disclosed herein. Thespecification should be considered exemplary only, and the scope of theinvention is accordingly intended to be limited only by the followingclaims.

1. A computer-implemented method of identifying the existence andposition of text in visual media content and determining a subject'slevel of interaction with the identified text, the method comprising:identifying a portion of visual media content that constitutes text;determining a position of the identified text of the visual mediacontent; presenting the visual media content to a subject; collectingeye data from the subject while the subject is viewing the visual mediacontent, the eye data including pupil data, blink data, and gaze data;generating a gaze pattern of the subject based on the collected eyedata; determining whether the subject has interacted with the identifiedtext based at least on the determined position of the identified textand the generated gaze pattern; and in response to a determination thatthe subject has interacted with the identified text: determining thesubject's level of interaction with the identified text; and (ii)determining the subject's emotional response to the identified text. 2.The computer-implemented method of claim 1, wherein determining thesubject's level of interaction with the identified text furthercomprises: determining whether the subject has read the identified text;and determining whether the subject has comprehended the identifiedtext.
 3. The computer-implemented method of claim 1, further comprising:identifying one or more characteristics associated with geometry of theidentified text.
 4. A computer-implemented system for identifying theexistence and position of text in visual media content and determining asubject's level of interaction with the identified text, the systemcomprising a computer-readable storage medium, the computer-readablestorage medium comprising one or more computer-readable instructionswhich when executed by a processor cause the processor to: identify aportion of visual media content that constitutes text; determine aposition of the identified text of the visual media content; present thevisual media content to a subject; collect eye data from the subjectwhile the subject is viewing the visual media content, the eye dataincluding pupil data, blink data, and gaze data; generate a gaze patternof the subject based on the collected eye data; determine whether thesubject has interacted with the identified text based at least on thedetermined position of the identified text and the generated gazepattern; and in response to a determination that the subject hasinteracted with the identified text: determine the subject's level ofinteraction with the identified text; and (ii) determine the subject'semotional response to the identified text.
 5. The computer-implementedsystem of claim 4, wherein the one or more computer-readableinstructions causing the processor to determine the subject's level ofinteraction with the identified text further include instructionscausing the processor to: determine whether the subject has read theidentified text; and determine whether the subject has comprehended theidentified text.
 6. The computer-implemented system of claim 4, whereinthe one or more computer-readable instructions further compriseinstructions causing the processor to: identify one or morecharacteristics associated with geometry of the identified text.